feat(ci): deterministic Odoo fixture + full verified/contradicted cov… - #753
Conversation
|
Reviewed and refreshed — Verified on the new head
The check I care about most passed
Mutation-checked to be sure — forcing Cost, since this corpus is the one that just started actually runningI costed these evals recently, and got it wrong the first time by reading corpus size as call count. For anyone doing the same arithmetic here: the two new files are 2 entries, not 70 — the rest of those line counts is (excellent) commentary. The one thing worth naming explicitly, because "no new env-var" in the description is true but is not the whole cost story: folding No interaction with #748 (the One nitThe description says the blast radius is No blockers from me. |
What
Golden-set eval v2 (Closes #639). Inject FixtureOdooReader into DeterministicChecker so a corpus entry declares frozen Odoo records the real checker re-queries — closes Gap 1 (ERP-confirmed hard claim → verified → approved) and Gap 2 (ERP-refuted → blocked). expected.via asserts the path so an entry can't pass for the wrong reason. Gap 3 (full-turn eval) deferred.
Test plan
Risk / blast radius
CI-only (test/golden/** + golden-eval.yml); no runtime/schema/API change, no new env-var.
Need help on this PR? Tag
@codesmith-botwith what you need. Autofix is disabled.